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ABSTRACT 


This  thesis  explores  alternatives  to  current  testing  methodology  being  applied  to 
two  TOW  missile  training  systems.  This  thesis  contends  that  current  practices  do  not 
adequately  prove  system  accuracy  or  system  training  value.  Research  emphasis  is  placed 
upon  identifying  those  factors  involved  in  assessing  system  accuracy  currently  being 
overlooked.  The  objective  is  that  future  government  testing  will  address  system  accuracy 
and  training  value  in  detail.  Following  a  description  of  current  techniques,  an  alternative 
to  current  accuracy  assessment  is  presented  using  the  precepts  of  direct  fire  gunnery  based 
upon  a  series  of  statistical  treatments  that  quantify  system  accuracy  and  contract 
specification  compliance.  Data  collection  enhancements,  potential  test  design 
modifications,  and  a  methodical  data  analysis  plan  is  presented.  An  alternative  testing 
scenario  is  developed  based  upon  the  recommended  changes  in  test  methodology. 
Finally,  observations  and  recommendations  are  provided  pertaining  to  program 
management  of  the  two  TOW  missile  training  systems  in  an  effort  to  optimize  program 
structure.  The  underlying  premise  is  that  the  application  of  operations  research  skills  to 
validate  system  performance  will  improve  the  final  product  fielded  to  U.S.  Marines  and 


Soldiers. 
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EXECUTIVE  SUMMARY 


The  pursuit  of  realistic  training  has  taken  on  greater  importance  as  the  price  of 
munitions  and  the  cost  of  using  live-fire  training  ranges  has  risen  dramatically.  In  1986, 
the  U.S.  Army  and  U.S.  Marine  Corps  contracted  to  develop  a  Precision  Gunnery 
Training  System  (PGTS)  that  would  permit  inexpensive  training  of  Tube  Launched, 
Optically  Tracked,  Wire  Guided  (TOW)  gunners.  Following  recent  technological 
advances  in  thermal  imagery  and  computer  processing  power,  enhancements  to  the  PGTS 
program  have  been  pursued  to  match  the  advancements  being  made  in  the  TOW  tactical 
system.  Two  resulting  PGTS  programs,  currently  under  development,  are  the  focus  of 
this  thesis.  The  subject  programs  are  the  Enhanced  Instractors  Station,  commonly 
referred  to  as  REHOST,  and  the  Tactical  Engagement  System  (TES). 

This  thesis,  based  upon  a  seven-week  internship  with  the  prime  PGTS  contractor, 
questions  the  validity  of  the  training  value  of  the  REHOST  program  and  the  adequacy  of 
the  current  approach  to  assessing  the  TES  system’s  accuracy.  Neither  issue  is  adequately 
addressed  in  program  documentation  which  leads  to  the  conclusion  that  sound 
procurement  decisions  cannot  be  made  at  this  point  in  the  procurement  cycle.  The  intent 
of  this  thesis  is  to  provide  alternatives  to  current  testing  methodology  that  might  enhance 
future  testing  evolutions. 

Following  a  description  of  the  currently-fielded  TOW  system  and  PGTS  training 
functions,  the  Improved  Tactical  Acquisition  System  (ITAS)  is  introduced  as  the  new 
host  system  of  the  PGTS  family.  It  is  the  development  of  ITAS  technology  that  is  driving 
TES  procurement.  The  enhanced  thermal  and  autotrack  capabilities  of  the  ITAS  require  a 
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modification  of  the  PGTS  family  in  order  to  maintain  realism  in  training. 

The  current  approach  to  program  testing  is  highlighted  as  lacking  in  data 
collection  and  contract  specification  compliance.  Three  weeks  of  observed  testing  at 
Redstone  Arsenal  serves  as  the  basis  of  the  testing  research  presented  herein. 
Recommendations  are  made  to  enhance  the  current  testing  process  in  an  effort  to  establish 
quantitative  measures  of  TES  accuracy.  A  similar  set  of  recommendations  is  made 
regarding  REHOST  specification  compliance. 

Having  provided  an  alternative  to  current  testing  practices,  the  thesis  recommends 
an  analytical  approach  to  assessing  TES  system  accuracy.  A  scenario  is  developed  that 
utilizes  the  proposed  alternative  to  current  testing  practices.  Emphasis  is  placed  upon 
redesigning  the  data  collection  plan  associated  with  this  effort  and  identifying  potential 
categories  of  variability  affecting  system  accuracy.  Several  statistical  treatments  are  then 
proposed  in  an  effort  to  establish  quantitative  measures  of  effectiveness  to  be  used  in 
assessing  TES  accuracy.  Probability-of-hit  upon  a  circular  target,  multiple  regression, 
hypothesis  testing,  a  binomial  distribution  assessment,  and  nonparametric  alternatives  are 
provided  as  potential  alternatives  to  the  “hit  or  miss”  approach  currently  in  use.  An 
example  scenario,  using  simulated  data,  is  then  conducted  implementing  the  use  of 
hypothesis  testing  and  confidence  intervals  in  assessing  system  accuracy. 

Finally,  the  REHOST  system’s  suitability  as  a  meaningful  training  system  is 
assessed.  Lack  of  concurrent  development  of  tactical  and  training  systems  and  the  fact 
that  the  REHOST  system  is  not  based  upon  a  fielded  tactical  system  are  emphasized  as 
major  contributors  to  a  current  lack  of  realism  in  the  REHOST  system.  Future  research  is 
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recommended  to  explore  the  correlation  between  gunner  performance  using  the  REHOST 
system  and  similar  performance  using  fielded  tactical  systems. 

Fielding  quality  training  or  tactical  systems  requires  innovation  in  today’s  austere 
budget  environment.  Specific  measures  of  effectiveness  must  be  required  throughout  the 
procurement  process.  No  knowledgeable  decision  may  be  made  without  some 
quantifiable  measure  of  performance.  Therefore,  it  would  not  be  prudent  to  make  a 
procurement  decision  on  the  PGTS  systems  based  solely  upon  “hit  or  miss”  testing  in  the 
case  of  the  TES  system,  or  prior  to  some  determination  of  training  value  in  the  case  of  the 
REHOST  system. 
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I.  INTRODUCTION 


The  March  1996  version  of  the  DoD  5000  series  of  procurement 
guidelines  “ requires  an  acquisition  environment  that  makes  DoD  the 
smartest,  most  responsive  buyer  of  the  best  goods  and  services,  that  meet  our 
warfighters’  needs  at  the  best  dollar  value  over  the  life  of  the  product  ” 
fReflJ 

A.  BACKGROUND 

The  pursuit  of  realistic  training  has  taken  on  greater  importance  as  the  price  of 
munitions  and  the  costs  involved  in  the  use  of  live-fire  training  ranges  have  risen 
dramatically.  Simultaneous  to  the  rise  in  weapons  costs,  pressures  on  defense  spending 
have  increased  as  well.  The  U.S,  Army  and  the  U.S.  Marine  Corps  in  1986  sought  to 
address  this  problem  as  it  applies  to  Tube  Launched,  Optically  Tracked,  Wire  Guided 
(TOW)  missile  training  with  the  development  of  the  Precision  Gunnery  Training  System 
(PGTS).  The  PGTS  program  was  initiated  to  address  tactical  anti-armor  training  without 
the  use  of  live-fire  ranges  or  the  expenditure  of  live  ammunition.  Through  mid  1995, 
deliveries  of  approximately  450  Field  Tactical  Trainers  (FTTs),  approximately  350 
Gunnery  Trainers  (GTs)  and  approximately  275  vehicle  mounted  systems  had  been 
fielded  to  soldiers  and  marines  at  cumulative  contract  prices  exceeding  85  million  dollars. 
[Ref  2] 

Two  recent  developments  have  brought  forth  an  initiative  to  improve  upon  these 
currently  fielded  training  systems.  The  first  development  is  the  advancement  of 
technology  in  the  areas  of  computer  processing  power  and  graphical  user  interfaces.  The 
second  development  is  a  U.S.  Army  initiative  to  field  a  second  generation  Forward 
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Looking  Infrared  (FLIR)  sight  for  the  TOW  tactical  system  through  an  Advanced 
Concepts  Technology  Demonstration  (ACTD)  program. 

B.  CHALLENGE 

The  effective,  timely  and  concurrent  development  of  tactical  and  training  systems 
demands  a  new  approach  to  addressing  program  issues.  This  thesis  proposes  that  a 
mamage  of  operations  research  skills  and  recent  acquisition  reform  initiatives  constitute 
the  basis  for  success.  Acquisition  of  quality  weapons  systems  requires  constant 
consideration  of  the  operational  objectives  intended  to  be  met  by  a  given  procurement. 
The  majority  of  development  effort  must  be  applied  to  developing  measures  of 
effectiveness  that  can  be  quantified  so  fiiat  program  decision  makers  might  make 
informed  procurement  decisions.  The  combined  talents  of  management,  Integrated 
Product  Teams  (IPTs),  and  analysts  capable  of  quantifying  weapon  system  performance 
and  cost  parameters  must  be  applied  in  tandem  to  ensure  the  successful  accomplishment 
of  this  requirement. 

C.  OBJECTIVES 

This  thesis  is  based  upon  the  author’s  participation  in  a  seven  week  internship  at 
the  prime  contractor’s  PGTS  program  headquarters.  It  will  suggest  an  alternative 
approach  to  assessing  the  accuracy  of  the  Field  Tactical  Trainer,  referred  to  in  this  thesis 
as  the  Tactical  Engagement  Simulation  (TES)  mode  of  the  PGTS  family.  An  alternative 
to  current  testing  methodology  being  utilized  in  this  program  is  presented  and,  as  a 
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secondary  thesis  objective,  programmatic  issues  surrounding  the  indoor  Guimery  Trainer, 
commonly  referred  to  as  the  REHOST  improved  instructor’s  station  are  addressed.  It  is 
hoped  that  the  testing  alternatives  presented  in  this  thesis  might  be  useful  in  future 
government  testing  efforts.  It  must  be  noted  that  the  PGTS  contractor’s  enthusiasm  for 
input  from  an  operational  user  and  the  ability  to  access  all  program  areas  greatly 
facilitated  the  author’s  quest  to  meet  these  thesis  objectives. 

The  thesis  is  organized  into  the  following  areas; 

♦  An  introduction  to  the  currently  fielded  TOW  tactical  and  PGTS  training 
systems  is  provided. 

♦  Enhancements  that  are  currently  being  developed  are  introduced. 

♦  Current  techniques  being  utilized  to  assess  the  effectiveness  of  the  developing 
systems  are  described. 

♦  An  alternative  to  current  testing  techniques  is  presented. 

♦  A  test  scenario  using  the  above  alternative  test  methodology  is  provided. 

♦  Finally,  program  observations  and  recommendations  are  made  pertaining  to 
the  subject  training  systems. 

The  approach  taken  to  address  the  above  areas  consists  of  a  description  of  the 
current  process  followed  by  recommendations  to  improve  that  process.  Emphasis  is 
placed  upon  quantifying  the  accuracy  parameters  described  in  Chapters  III  through  V  in 
order  to  prove  that  the  training  systems  meet  the  requirements  set  forth  in  Reference  2. 
Due  to  the  author’s  repeated  involvement  in  the  procurement  process,  observations  and 
recommendations  are  also  made  pertaining  to  the  management  and  organizational 
practices  of  government  and  contractor  program  offices. 
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n.  PROGRAM  DESCRIPTION 


A.  CURRENT  SYSTEMS 


Figure  1:  The  current  TOW  system  in  its  ground  mount  configuration.  Both  the  day  and  night 
thermal  viewer  are  visible  atop  the  TOW  system,  while  the  Fire  Control  Subsystem  (FCS)  is  placed 
below  the  tripod.  The  source  of  this  photo  is  Reference  3. 


I  To  begin  discussion  involving  the  PGTS  training  systems  being  developed,  one 

must  possess  an  understanding  of  the  tactical  system  that  PGTS  is  designed  to  emulate. 
The  TOW  system  is  the  primary  anti-armor  weapon  of  the  infantry  forces  in  the  U.S. 
Army  and  the  U.S.  Marine  Corps.  The  current  tactical  TOW  missile  system  is  deployed 

1 

on  a  variety  of  platforms.  It  is  shown  in  Figure  1  in  its  ground  mount  configuration.  The 
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configuration.  The  system  provides  the  gunner  with  two  alternative  optical  devices  to 
acquire  and  engage  targets.  One  sight  provides  daylight  engagement  capability  while  the 
other  provides  passive  technology  that  permits  limited  visibility  engagements  during 
acceptable  ambient  light  conditions.  The  system  must  align  a  thermal  or  xenon  beacon 
with  a  signal  receptor  in  the  daylight  sight.  The  missile  beacon  and  signal  receptor  are 
likewise  aligned  with  the  day  reticle  or  aimpoint.  The  thermal  viewer  is  then  aligned  to 
the  day  sight.  This  boresighting  or  alignment  process  establishes  baseline  system 
accuracy.  Throughout  any  engagement,  the  TOW  gunner  must  manually  guide  the  wire- 
guided  TOW  missile  to  the  intended  target  by  using  the  hand  grips  of  the  TOW  traversing 
unit.  The  traversing  imit  rests  upon  the  tripod  visible  in  Figure  1  and  sends  correction 
signals  to  the  missile  via  the  wire  guide.  In  order  to  ensure  that  the  target  is  within  the 
maximum  3750  meter  range  capability  of  the  TOW  system,  the  crew  must  currently 
obtain  range-to-target  information  from  an  outside  source.  Recent  missile  advances 
permit  top-down  attack  in  addition  to  more  conventional  side  aspect  engagements.  The 
current  POTS  family  of  training  devices  attempts  to  emulate  the  TOW  missile  flight 
characteristics  within  the  ITAS  optical  system  for  use  by  field  units  conducting  TOW 
guimery  training  where  live  ordnance  is  not  deemed  efficient. 
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B.  OUTDOOR  TRAINING  ENHANCEMENTS 


Figure  2:  The  Improved  Target  Acquisition  System  (TTAS)  featuring  the  integrated  Day  /  Night 
viewer.  The  FCS  shown  is  similar  to  that  in  Figure  1.  The  source  of  this  photo  is  Reference  4. 


Due  to  the  development  of  the  ITAS,  shown  in  Figure  2,  an  effort  to  develop 
compatible  PGTS  training  systems  has  commenced.  The  goal  is  to  provide  equivalent  or 
enhanced  capabilities  over  the  current  PGTS  family. 

The  major  improvements  to  the  PGTS  outdoor  training  systems  being  undertaken 
by  the  training  system  developer  are  threefold:  [Ref.  5] 
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1.  An  Embedded  Training  (ET)  mode  will  permit  the  gunner  to  fly  simulated 
TOW  missiles  against  targets  of  opportunity  in  the  training  area  being  utilized.  ET  is  a 
single  circuit  card  that  fits  into  the  current  jSre  control  system  (PCS),  also  shown  in 
Figure  2.  It  permits  the  gunner  to  fly  a  simulated  missile  while  looking  through  the 
integrated  sight  in  thermal  mode  with  or  without  autotrack  engaged.  The  ET 
enhancement  has  progressed  through  initial  government  procurement.  The  ET  system  is 
not  a  subject  of  interest  for  this  thesis  but  this  system  does  represent  a  new  training 
feature  over  current  PGTS  systems. 

2.  A  Tactical  Engagement  Simulation  (TES)  mode  builds  upon  the  ET  mode  and 
permits  a  gunner  to  engage  targets  of  opportunity  on  the  training  battlefield  and  receive 
performance  feedback  via  the  Multiple  Integrated  Laser  Engagement  System  (MILES  II) 
training  system.  MILES  provides  an  opportunity  for  most  weapons  systems  in  the  U.S. 
inventory  to  engage  one  another  by  emitting  a  coded  laser  burst  that  is  received  and 
returned  by  receptor  belts  attached  to  a  target  system.  The  laser  code  or  intensity  of  the 
pulse  received  by  the  target  determines  whether  the  engagement  result  is  a  kill,  miss,  or 
some  other  less  catastrophic  degree  of  battle  damage.  The  TES  training  system  is  also  a 
new  capability  over  existing  training  systems  and  will  be  the  primary  focus  of  this  thesis. 

3.  The  Precision  Gunnery  (PG)  mode  will  enable  the  gimner  to  be  evaluated  in 
terms  of  target  tracking  proficiency.  The  PG  mode  will  be  the  ITAS  equivalent  to  current 
TOW  guimery  evaluation  capability. 

All  three  of  these  enhancements  will  be  developed  for  use  with  the  TOW 
Improved  Target  Acquisition  System  (ITAS).  This  system  represents  the  latest  FT  TP 
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technology  that  is  replacing  the  current  optical  and  thermal  sights  with  an  integrated  sight 
capable  of  day  and  limited  visibility  engagements  through  the  same  ocular.  FUR 
development  is  proceeding  imder  a  separate  contract  by  a  different  prime  contractor. 
Enhancements  of  interest  to  the  current  TOW  tactical  gunnery  system  under  the  ITAS 
program  include: 

♦  Increased  target  detection  ranges  under  all  visibility  conditions 

♦  Autotracking  of  targets  in  thermal  mode 

♦  Organic  rangefinding  capability.  [Ref  6] 

These  enhancements  are  exceptional  and  will  bring  the  TOW  gunner  effectively  to 
the  21st  century  battlefield.  Autotracking  of  potential  targets,  organic  rangefinding  and 
improved  limited  visibility  optics  will  greatly  enhance  this  weapon’s  effectiveness  in  the 
hands  of  a  capable  crew.  Similarly,  the  enhancements  being  made  to  the  PGTS  training 
systems  do  not  impact  upon  the  performance  of  the  tactical  system  while  engaged  in  live 
fire.  The  PGTS  is  therefore  designed  to  remain  passive  in  the  TOW  ITAS  fire  control 
system  until  called  into  use  by  the  crew.  The  ultimate  goal  is  to  possess  an  integrated 
tactical  and  training  capability. 

C.  INDOOR  TRAINING  ENHANCEMENTS 

The  enhancements  described  to  this  point  all  pertain  to  the  TOW  gunnery  system 
used  in  the  field  environment.  A  final  enhancement  to  the  PGTS  family  and  the 
secondary  subject  of  this  thesis  is  the  improved  instructor’s  station,  or  gunnery  trainer, 
used  during  indoor  training  and  shown  in  Figure  3.  This  program  is  known  as  the 
REHOST  program  and  is  designed  to  upgrade  the  current  video-based  training  system. 
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This  training  system  permits  TOW  gnnners  to  engage  video  targets  displayed  in  a  mock 
TOW  sighting  system.  Audio  and  visual  effects  are  provided  throughout  the  engagement 
scenario.  The  upgrade  being  performed  by  the  contractor  consists  of  a  computer  upgrade 
to  the  Pentium  processor  level  and  the  capability  that  permits  instructor  programming  of 
threat  scenarios  for  the  student.  [Ref.  7] 


Figure  3:  The  REHOST  system  with  the  instructor  station  (right)  and  the  student  station  Geft). 
Notice  the  difference  in  configuration  between  the  student  station,  shown  here,  and  the  TOW 
system  in  Figure  1.  The  source  of  this  photo  is  Reference  7. 


The  PGTS  program  is  being  executed  imder  the  original  1986  contract  as 
individual  contract  modifications  for  each  of  the  major  subsystems:  Tactical  Engagement 
Simulation  (TES),  Embedded  Training  (ET),  Precision  Gunnery  (PG),  and  REHOST. 
ET,  TES  and  REHOST  are  preparing  for  government  testing  phases  and  potential 
production  contract  award  while  the  PG  program  remains  in  development. 
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m.  CURRENT  TESTING  METHODOLOGY 

“Test  and  evaluation  programs  shall  be  structured  to  provide  essential 
information  to  decision  makers,  assess  attainment  of  technical  performance 
parameters,  and  determine  whether  systems  are  operationally  effective,  suitable, 
and  survivable  for  intended  use. "  [Ref.  8] 

A.  THE  TEST  PLAN 

The  Redstone  Arsenal  in  Hunstville,  Alabama  was  the  scene  of  primary  testing 
for  validation  of  TES  system  performance  and  the  contractor’s  Trainer  Test  Procedures. 
The  Redstone  government  test  range  had  been  obtained  by  the  contractor  through  an 
agreement  with  the  government  program  developer.  The  results  of  this  test  phase  would 
provide  an  indication  of  system  maturity  as  the  contractor  was  preparing  to  sell  six  or 
seven  prototype  TES  systems  to  the  government.  The  procedures  used  to  guide  the 
testing  process  were  exceptionally  detailed  for  this  stage  of  the  development  process  due 
primarily  to  the  fact  that  they  had  been  modeled  from  the  earlier  development  effort  of  the 
ET  system  that  had  been  previously  assessed  through  the  government’s  acceptance 
process.  The  pertinent  sections  of  these  trainer  test  procedures  are  included  in  the 
Appendix.  The  approach  to  assessing  the  current  testing  process  began  with  a  study  of 
the  contract  under  which  the  contractor  was  performing.  Due  to  the  fact  that  this  1993 
contract  was  a  relatively  simple  modification  to  the  original  1986  document,  the  study  of 
the  1993  version  of  contract  specifications  to  be  tested  in  Redstone  was  completed 
expeditiously.  Subsequent  to  the  contract  specification  review,  a  review  of  the  draft  test 
procedures  was  conducted  in  an  effort  to  determine  whether  a  complete  matrix  consisting 
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of  contract  requirements  versus  test  procedures  existed.  One  should  anticipate  a  direct 
correlation  between  contract  requirements  and  test  procedures.  This  correlation  exists  in 
the  TES  program,  however,  the  lack  of  detailed  specifications  regarding  system  accuracy 
is  a  cause  for  concern.  [Ref.  9] 

B.  INITIAL  TEST  EXECUTION 

The  testing  at  Redstone  Arsenal  commenced  with  a  detailed  series  of  diagnostic 
tests  upon  the  TES  system  conducted  by  a  systems  engineer,  one  or  two  software 
engineers,  a  quality  assurance  representative  and  the  author.  The  use  of  a  diagnostic 
terminal  that  passively  observed  system  functions  provided  the  window  through  which 
performance  was  to  be  evaluated.  Information  passed  to  this  diagnostic  terminal  by  the 
ITAS  system  included  trigger  pull,  missile  launch,  tracking  rate  information  and  MILES 
engagement  code  information.  Throughout  the  testing  process,  as  system  malfunctions 
occurred.  Electronically  Programmable  Read  Only  Memory  (EPROM)  chips  could  be 
recoded,  thereby  permitting  updated  software  inputs  that  control  TES  simulated  missile 
flight.  The  software  troubleshooting  process  required  daily,  round-tiip  Federal  Express 
shipment  of  EPROMs  between  Huntsville,  Alabama  and  Long  Island,  New  York  with 
each  modification,  since  the  capability  to  “bum”  new  code  onto  the  EPROMs  was  not 
resident  at  Huntsville.  This  long  distance  process  was  time-consuming  and  not  conducive 
to  effective  range  utilization.  The  software  modification  process  also  revealed  numerous 
software  shortcomings  in  the  system  that  indicated  further  lab  work  would  be  beneficial 
prior  to  field  testing.  The  software  problems  encountered  were  not  unusual  for  this  stage 
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of  system  development,  however,  the  frequency  of  the  software  problems  point  to  a  lack 
of  system  readiness  for  the  field. 

This  initial  phase  of  testing  also  revealed  shortfalls  in  the  communication  network 
between  the  two  prime  contractors  involved,  and  the  government.  Due  to  the  proprietary 
nature  of  information  regarding  the  HAS  tactical  system  and  the  TES  training  subsystem, 
cross-communication  of  system  design  features  was  extreinely  limited.  Although  a 
technical  representative  from  the  FTAS  contractor  was  available  for  troubleshooting  the 
host  system,  all  fire  control  data  used  in  the  TES  training  subsystem  came  through  a  data 
bus,  which  is  standard  on  most  military  systems.  However,  the  true  meaning  of 
information  being  passed  to  the  data  bus  was  not  always  clear.  The  data  bus  serves  as  a 
common  access  point  to  system  performance  information  that  has  previously  been 
designated  as  necessary  information  for  system  integration.  This  method  of  information 
exchange  provides  that  data  which  is  necessary  for  contractors  to  perform  specified  tasks 
while  retaining  the  proprietary  design  rights  of  the  original  prime  contractor.  The 
importance  of  a  detailed  Interface  Control  Document  (ICD)  between  contractors  using  the 
common  data  bus  became  evident.  A  program  management  Integrated  Product  Team 
(IPT)  could  serve  this  purpose  well  and  lead  to  a  joint  development  effort  of  tactical 
(IT AS)  and  training  (PGTS)  systems. 

C.  CURRENT  ACCURACY  ASSESSMENT  TECHNIQUE 

The  method  utilized  in  assessing  system  accuracy  at  Redstone  Arsenal  consisted 
of  engaging  a  High  Mobility,  Multi-Purposed,  Wheeled  Vehicle  (HMMWV)  fitted  with 
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the  Multiple  hitegrated  Laser  Engagement  System  (MILES)  system.  The  ranges  varied 
from  zero  to  4000  meters  in  thermal  and  day  sight  mode.  Once  a  target  was  engaged,  the 
MILES  returns  were  interpreted  and  another  engagement  began.  Limited  records  of  these 
engagements  were  maintained.  The  author  attempted  to  capture  the  majority  of  available 
data  regarding  results  of  the  engagements.  The  results  of  this  data  collection  effort  are 
provided  in  Table  1.  At  ranges  of  500  to  3750  meters,  scoring  event  probabilities 
calculated  from  this  data  ranged  from  .67  to  .80,  significantly  below  the  contract  required 
.95  probability  of  scoring  event  occurrence.  [Ref.  6]  A  scoring  event  is  defined  as  any 
indication  that  the  MILES  laser  has  impacted  any  portion  of  the  engaged  target.  The 
scoring  event  reference  is  specific  to  the  MILES  system  of  target  engagement  using  the 
Blast  Laser  Transmitter  (BLT).  A  MILES  engagement  occurs  when  the  BLT  sends  a 
wide  beam  pulse  at  a  target  fitted  with  laser  receptors.  The  intention  is  to  return  a  signal 
to  the  gunner  when  a  target  is  successfully  engaged.  A  scoring  event  does  not  necessarily 
translate  to  a  MILES  “kill.”  A  kill  result  is  dependent  on  the  strength  of  the  laser  pulse 
received  by  the  target  receptors,  and  therefore,  places  a  premium  upon  system  accuracy. 
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Table  1:  TES  engagement  data  coUected  at  Redstone  Arsenal 
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D.  INITIAL  SPECIFICATION  AND  DATA  CONCERNS 

Following  the  observation  of  current  testing  and  a  review  of  the  contract 
specifications,  concerns  regarding  the  definition  of  realistic  performance  parameters  and 
adequate  data  collection  to  verify  those  parameters  must  be  addressed.  For  example,  the 
reliability  performance  parameter  of  the  system  is  only  encumbered  by  the  requirement 
that  any  embedded  subsystem  shall  not  degrade  the  ITAS  tactical  system’s  reliability. 
[Ref.  5]  This  implies  that  the  TES  mode  must  have  a  reliability  of  100%.  The  event  of 
100%  reliability  of  any  system  is  highly  unlikely.  Even  though  TES  spends  the  majority 
of  time  in  a  passive  mode,  when  in  use,  it  does  interact  with  the  HAS  system  via  the  Fire 
Control  Subsystem,  and  thus  presents  a  failure  possibility.  Specifications  and  accuracy 
requirements  that  necessitate  a  .95  probability  that  a  scoring  event  occurs,  lead  to  two 
system  design  problems.  First,  without  the  knowledge  of  system  aimpoint,  no  means 
exist  to  gather  miss  distance  data.  Secondly,  in  the  thermal  mode,  the  tendency  of  any 
gunner  will  be  to  engage  “center  of  visible  mass.”  In  the  thermal  mode,  this  will 
normally  be  the  center  of  the  “hot  spot”  of  the  target  to  be  engaged.  In  autotrack  mode, 
the  HAS  is  determining  this  spot  through  a  sampling  of  target  temperatures.  [Ref.  6] 
Although  effective  for  engaging  targets  with  live  ordnance,  this  situation  will  have  the 
gunner  engaging  a  point  on  the  target  that  may  not  necessarily  be  the  MILES  center  of 
visible  mass.  The  result  is  a  potential  bias  in  the  results  of  thermal  engagements  using  the 
TES  system  with  the  HAS  in  autotrack  mode.  Although  the  HAS  boresight  and 
autotrack  procedures  leading  to  system  aimpoint  are  proprietary,  the  data  regarding  the 
final  location  of  the  HAS  reticle  (system  aimpoint)  and  any  updates  performed  by  the 


15 


ETAS  system  is  critical  to  assessing  proper  TES  training  system  functioning.  The  lack  of 
available  data  severely  limits  the  analytical  approaches  available  in  assessing  system 
performance.  Table  1  provides  an  example  of  the  cumulative  data  for  a  week  of  testing  at 
Redstone  Arsenal.  It  is  indicative  of  the  lack  of  an  adequate  data  collection  plan 
supportive  of  analysis  in  determining  system  performance. 

The  lack  of  a  data  collection  effort  made  it  apparent  that  any  parametric  analysis 
of  system  accuracy  would  be  conducted  only  through  simulation.  For  this  reason,  the 
author’s  research  efforts  are  focused  upon  developing  an  alternative  to  the  current  test 
methodology  being  utilized. 
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IV.  AN  ALTERNATIVE  TEST  METHODOLOGY 


A.  INTRODUCTION 

Prior  to  the  start  of  any  testing,  it  is  imperative  that  the  goals  of  testing  be 
established  well  before  the  first  day  on  the  test  range  or  in  the  laboratory.  The  ability  to 
define  factors  affecting  system  performance  must  drive  test  design.  For  example,  an 
appropriate  test  of  system  accuracy  must  begin  with  defining  those  variables,  or  biases, 
that  impact  upon  system  accuracy.  It  is  through  this  type  of  testing  process,  whether 
government  or  contractor  sponsored,  that  system  maturity  is  attained.  A  test-fix-test 
approach  encourages  a  systematic  troubleshooting  process  leading  to  the  development  of 
a  system  that  meets  a  stated  mission  need.  The  intent  of  this  chapter  is  to  provide  an 
alternative  testing  methodology  focused  primarily  upon  assessing  accuracy  of  the  TES 
system,  but  with  applications  to  REHOST  as  well.  Again,  the  hope  is  that  this  proposal 
will  benefit  future  testing  efforts.  It  is  not  feasible  to  address  all  aspects  of  system  testing 
in  their  entirety  here.  The  emphasis  will  be  placed  upon  assessing  system  accuracy,  but 
this  approach  does  have  applications  to  ancillary  test  areas  such  as  environmental,  shock, 
and  reliability  testing. 

B.  A  PROPOSED  TEST  METHODOLOGY 

Initial  emphasis  in  approaching  testing  is  best  directed  toward  establishing  a  core 
group  of  trained  test  personnel.  This  core  group,  operating  as  a  distinct  testing-oriented 
IPT,  could  be  resident  in  the  contractor’s  quality  assurance  section.  As  programs  progress 
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and  program  personnel  gain  experience  in  testing,  expertise  will  eventually  be  resident 
throughout  the  organization.  This  approach  is  suitable  to  programs  of  any  degree  of 
complexity.  It  is  sufficiently  generic  so  that  a  program  manager  can  easily  tailor  his  test 
team  to  meet  his  system’s  test  requirements.  Aside  from  the  literature  available  on  the 
testing  process,  training  in  test  execution  can  be  obtained  through  the  following  means: 

♦  Defense  Systems  Management  College  (DSMC)  provides  short  courses  such  as 
the  two-week  Test  and  Evaluation  Management  Course  (TEMC)  and  the  full  Program 
Managers  course.  Contractor  participation  is  normally  welcome,  as  it  fosters  the 
govemment/industry  team  concept. 

♦  Government  test  agencies  can  also  provide  a  wealth  of  testing  knowledge.  The 
corporate  knowledge  resident  at  such  agencies  as  the  Test  and  Evaluation  Command 
(TECOM),  the  Army  Materiel  Systems  Analysis  Agency  (AMSAA),  and  a  host  of  similar 
organizations  can  be  of  assistance  in  executing  any  test  evolution. 

♦  Government  program  offices  will  readily  share  their  testing  experiences  in 
pursuit  of  fielding  superior  systems.  Assistance  lent  by  the  government  early  in  program 
development  will  facilitate  the  government  testing  phase  prior  to  production  contract 
award. 

♦  Historical  documentation  of  previous  programs  can  also  provide  useful  guides 
to  testing.  The  use  of  previous  Test  and  Evaluation  Master  Plans  (TEMPs)  and  Data 
Collection  and  Analysis  plans  will  serve  to  provide  insights  into  test  design  concepts  as 
well  as  maintain  homogeneity  among  test  documents  where  necessary  and  convenient. 
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Prior  to  the  execution  of  testing,  a  thorough  systems  analysis  should  be  conducted. 
It  is  possible,  depending  upon  system  design,  that  simultaneous  testing  of  subsystems 
might  be  conducted.  This  parallel  approach  to  testing  can  save  scarce  resources. 
However,  the  same  systems  analysis  may  point  out  that  future  known  design  changes 
make  immediate  testing  of  a  particular  subsystem  fruitless  at  the  current  time.  Repetition 
of  test  events  due  to  system  design  changes,  particularly  in  software,  are  expensive  and 
time  consuming. 

This  systems  analysis  will  establish  a  critical  path  in  the  system  design  process. 
The  critical  path  identifies  those  events  that  are  dependent  upon  one  another  for 
completion  or  modification.  In  other  words,  critical  path  events  must  be  accomplished 
sequentially.  Once  identified,  the  critical  path  can  be  used  to  develop  the  sequence  of 
testing.  Critical  path  testing  events  are  scheduled  so  that  a  future  modification  in  design 
does  not  require  repetition  of  previous  tests.  Meanwhile,  subsystem  tests  not  on  the 
critical  path  should  be  available  for  execution  during  system  modification  periods  or 
unique  test  facility  availability  circumstances.  The  ability  for  program  managers  to  have 
“hip-pocket”  testing  events  ready  to  go  at  any  time  is  an  indicator  of  program  efficiency 
and  flexibility. 

Having  conducted  the  systems  analysis  and  defined  the  critical  path  of  testing,  it  is 
time  to  begin  identification  of  technical  aspects  of  the  testing  process.  This  process 
begins  with  the  contractually  binding  design  specifications  and  continues  through 
identification  of  data  items  in  support  of  meeting  requirements.  A  prime  example  is  the 
pursuit  of  achieving  a  contractual  degree  of  accuracy.  The  approach  to  meeting  accuracy 
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goals  can  be  achieved  in  a  myriad  of  ways.  Identifying  the  factors  affecting  accuracy  and 
a  test  design  that  measures  the  factors  of  interest,  must  go  hand  in  hand.  The  goal  of  the 
test  design  team  must  be  to  identify  the  variables,  both  dependent  and  independent,  and  a 
plan  to  fix,  control,  and  measure  those  variables  where  possible.  Although  the  tester 
cannot  hope  to  control  independent  variables  such  as  weather,  he  must  recognize  those 
that  impact  system  performance.  The  process  of  identifying  data  requirements  is  best 
started  by  identifying  the  essential  elements  of  analysis  (EEAs).  The  EEAs  serve  to 
provide  a  guide  to  required  data  collection  efforts  and  test  design.  Each  EEA  will  likely 
indicate  a  single  data  item  requirement  that  must  be  drawn  during  testing.  Each  of  the 
EEA’s  and  their  associated  data  items  may  then  form  an  overall  measure  of  effectiveness. 
In  the  case  of  assessing  system  accuracy,  each  EEA  will  identify  an  individual  data 
element  related  to  an  identified  bias  that  affects  system  accuracy.  These  individual  data 
elements  will  cumulatively  form  an  overall  accuracy  measure  of  effectiveness.  [Ref.  9] 
Without  this  approach,  the  accuracy  of  the  TES  system  cannot  be  determined. 

In  the  discussion  surrounding  the  TES  system,  direct  fire  variables,  or  biases,  are 
the  focus  of  this  research.  The  ability  to  identify  sources  of  variability  is  the  key  to 
effective  system  troubleshooting  and  repeatability  of  developmental  tests.  One  must  be 
able  to  develop  a  cause  and  effect  relationship  between  the  biases  affecting  accuracy  and 
the  probability-of-hit  of  the  tested  system  in  order  to  recreate  any  desired  test  scenario  and 
facilitate  identification  of  system  deficiencies.  Potential  methods  in  variable 
identification  are  discussed  in  the  error  budget  section  of  the  next  chapter.  The  thorough 
test  designer  will  recognize  that  effort  applied  to  this  area  will  impact  test  range 
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requirements,  instrumentation  requirements,  data  collection  plans  and  resource  allocation. 
Many  items  involved  in  testing  are  long-lead  items  requiring  early  identification. 
Examples  include  highly-skilled  test  personnel,  instrumented  test  ranges,  and  special 
ordnance,  to  name  a  few.  It  is  important  to  note  that  resource  allocation  is  not  limited  to 
dollars  alone.  Early  identification  of  required  test  personnel  with  specific  skills  will 
facilitate  efficient  testing.  Once  involved  in  testing,  the  sudden  recognition  of  a  need  for 
a  test  team  member  with  specific  skills  could  be  catastrophic  to  both  schedule  and  budget. 
Of  equal  importance  to  properly  planning  test  resources  is  the  technique  that  will  be  used 
to  develop  the  plan  to  analyze  any  data  collected.  If  statistical  treatments  are  to  be 
applied,  defining  that  approach  early  will  assist  in  data  collection  requirements, 
determination  of  sample  size  of  test  events,  and  procurement  of  prototypes.  The  answers 
provided  through  the  development  of  sample  sizes  and  test  duration  have  major 
budgetary,  as  well  as  schedule,  implications. 

The  majority  of  testing-related  discussion,  thus  far,  has  focused  upon  system 
accuracy.  The  methodology  being  addressed  in  this  chapter  will  apply  to  other  areas  of 
testing  as  well.  Reliability  testing,  which  is  often  classified  separately,  should  begin 
immediately  and  under  the  same  guidelines  discussed  above  in  terms  of  data  collection 
and  test  design.  Events  occurring  throughout  all  testing  provide  potential  reliability  data. 
This  can  be  as  simple  as  logging  operating  hours,  laser  firings  and  trigger  pulls,  or  as 
complex  as  conducting  failure  analysis  during  maintenance  periods.  Boresighting 
problems  involving  the  IT  AS  and  the  TES  provided  a  window  of  reliability  data- 
gathering  potential.  Maintaining  testing  focus  upon  those  areas  that  are  controllable  will 
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provide  data  that  is  understandable  and  easier  to  interpret.  Without  thorough 
understanding  of  the  biases  impacting  testing,  it  is  impossible  to  establish  the  meaning  of 
skewed  datapoints  or  outliers.  The  detailed  testing  approach  described  here  also  provides 
the  potential  to  explore  other  testing  opportunities  outside  the  terms  of  the  current 
contract,  whose  results  might  provide  insight  into  future  development  plans.  The 
potential  for  product  improvement  plans  or  follow-on  systems  may  be  unveiled  during 
testing  of  the  current  system  and  provide  the  beginnings  of  future  business.  The  litmus 
test  of  effective  testing  is  that  each  test  event  must  provide  a  result  that  is  meaningful  to 
the  program  manager  or  the  procurement  decision-maker. 

All  of  the  efforts  outlined  to  this  point  occur  before  the  first  test  event.  These 
preparatory  events  should  make  test  execution  easy  in  comparison  to  pre-test  preparation. 
Emphasis  upon  pre-testing  efforts  is  the  practical  approach  due  to  the  fact  that,  on 
complex  instramented  ranges,  an  hour  of  instrumented  test-time  can  translate  to 
thousands  of  dollars  of  testing  costs.  Once  testing  commences,  thoughts  must  turn  to 
effective  documentation  of  all  events  via  the  use  of  standardized  test  forms.  The  detailed 
chronology  of  test  events  is  not  a  substitute  for  detailed  data  collection  or  vice  versa.  The 
two  means  of  documenting  test  activity  should  be  complementary  in  nature.  The  testing 
chronology  will  include  details,  such  as  logistics  support  and  system  design  entries  not 
included  in  the  raw  data  collection  log.  The  testing  chronology  will  also  be  more  useful 
as  a  historical  document  in  support  of  future  testing.  Most  importantly,  it  will  be  an 
invaluable  tool  in  identifying  potential  outliers  in  the  data  gathered  and  the  reasons  for  the 
unusual  data  points. 
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Scheduling  of  test  activities  must  be  as  detailed  as  possible.  An  hourly  test 
schedule  will  not  only  facilitate  the  scheduling  of  test  personnel,  but  it  will  also  dissuade 
interruptions  from  outside  agencies  who  have  been  issued  a  copy  of  the  test  schedule. 
The  detailed  test  schedule  will  also  provide  a  guide  for  logistical  support  requirements 
and  program  management. 

The  final  phase  of  testing  is  post-testing  activities.  The  focus  here  must  be  upon 
timely  data  reduction  and  analysis.  This  will  be  discussed  in  detail  in  the  next  chapter. 
Potentially  sigmficant  post-test  activities  consist  of  recognition  of  support  in  testing  both 
inside  and  outside  the  test  organization.  Recognizing  superior  support  of  range  personnel 
will  reap  benefits  in  future  testing  evolutions  at  that  facility.  The  single  most  important 
post-test  event  is  the  after-action  report  of  testing.  This  report  will  be  less  detailed  than 
the  final  test  report  and  will  include  those  areas  of  the  test  that  went  well  or  require 
improvement  from  the  standpoint  of  test  execution.  The  after-action  debriefing  and 
report  of  the  EPT  will  serve  immeasurably  to  enhance  future  testing  evolutions.  Effective 
after-action  activities  will  enhance  test  documentation  procedures,  logistics  support  and 
test  scheduling.  As  described  in  this  chapter,  the  actual  effort  involved  in  system  testing 
is  small  in  comparison  to  the  efforts  that  take  place  before  and  after  a  successful  test 
evolution. 
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V.  A  NEW  APPROACH  TO  ASSESSING  TES  ACCURACY 


A.  roENlTFY  THE  VARIABLES 

In  order  to  adequately  address  the  accuracy  issue  in  the  TES  program,  an  error 
budget  is  developed  that  details  the  varibles  of  interest  that  affect  system  accuracy.  The 
TES  mode  of  operation  provides  a  period  of  simulated  missile  tracking,  and  then  at  the 
appropriate  time,  sends  a  burst  laser  transmission  to  the  target.  It  is  therefore  appropriate 
to  treat  TES  as  a  delayed  direct  fire  system.  The  direct  fire  approach  assists  in  the 
accuracy  factor  analysis  through  the  simplicities  associated  with  this  assumption.  The  use 
of  the  Blast  Laser  Transmitter  (BLT)  laser  also  simplifies  factor  analysis  in  that  many 
environmental  factors  such  as  crosswind,  are  si^ficantly  reduced,  as  well  as  certain 
weapon  system  influences,  such  as  tube  jump  or  round  to  round  dispersion.  Although  the 
TES  system  is  at  the  mercy  of  the  tactical  system  in  that  it  inherits  much  of  the  error 
budget  through  acceptance  of  the  IT  AS  system  boresight  and  autotrack  signals,  this  fact 
does  not  provide  relief  fi'om  addressing  other  potentially  significant  biases  such  as  BLT 
boresight  to  the  ITAS  tactical  system  and  BLT  aimpoint  biases  based  upon  engineering 
design  specifics.  Using  the  approach  outlined  in  Chapter  IV,  the  future  PGTS  testing 
efforts  might  address  the  following  as  significant  bias  categories: 

Cumulative  Inherited  Bias:  This  would  represent  the  bias  inherited  from  the  ITAS 
tactical  system.  It  would  have  to  be  derived  from  the  error  budget  associated  with  the 
ITAS  system. 
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Gunner  Aim  Error:  Numerous  gunners  were  employed  in  the  testing  process.  This 
provides  the  potential  for  variance  among  the  engagements  that  could  be  avoided.  One 
option  would  be  to  remove  the  gunner  from  the  equation  and  begin  with  stationary  system 
to  stationary  target  testing  to  establish  initial  accuracy  levels. 

Platform  Stability:  This  bias  refers  to  the  method  used  at  Redstone  Arsenal  of 
firing  from  a  mobile  trailer.  This  bias  is  easily  removed  by  moving  the  testing  to  a  firmer, 
more  solid  platform  such  as  a  concrete  platform. 

Target  belt  orientation:  The  target  belt  orientation  should  coincide  with  the  center 
of  thermal  mass.  This  should  assist  in  reducing  the  system  aimpoint  bias  associated  with 
ITAS  autotrack  engagements. 

BLT  boresight  retention:  This  is  potentially  the  largest  bias  directly  attributable  to 
the  TES  system.  This  bias  is  a  factor  in  the  ability  of  the  BLT  to  maintain  its  reference 
aimpoint  in  both  the  stationary  and  mobile  configuration.  No  current  test  effort  is  applied 
to  boresight  retention  following  movement  to  a  new  firing  position.  This  bias  warrants 
additional  consideration  in  future  testing  efforts.  [Ref  5] 

BLT  tube-to-tube  variation:  Similar  to  boresight  retention,  a  potential  bias  among 
direct  fire  systems  is  variation  among  individual  systems.  The  impact  of  this  bias  was 
observed  during  testing  at  Redstone.  Accuracy  biases  associated  with  BLT 
manufacturing  practices  will  Only  be  identified  through  additional  repeatable  testing. 

There  are  numerous  other  biases  that  would  normally  be  addressed  in  any  direct 
fire  weapon  system.  Ambient  temperature,  weather,  propellant  temperature  and 
crosswind  are  but  a  few.  These  are  not  interesting  in  the  TES  application  due  to  the  fact 
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that  not  all  of  these  biases  can  be  eliminated  nor  are  they  applicable  to  this  system.  The 
error  budget  will  serve  as  the  guide  that  identifies  any  system’s  pertinent  biases 
contributing  to  inaccuracy.  [Ref.  10] 

B.  STATISTICAL  OPTIONS  IN  ACCURACY  ANALYSIS 

The  fact  that  TES  performs  in  a  manner  similar  to  direct  fire  systems  permits 
application  of  relatively  simple  probability-of-hit  models  to  establish  system  accuracy. 
Although  this  is  a  positive  alternative  in  terms  of  accuracy  analysis,  application  of  the 
following  probability  of  hit  model  does  require  a  significant  modification  to  the  current 
approach  to  data  collection.  A  means  to  capture  information  on  chosen  miss  distance 
variables,  such  as  boresight  retention,  is  required.  The  results  of  a  statistical  treatment  of 
accuracy  data  addressing  the  major  bias  categories  detailed  previously,  will  lead  to  a  more 
realistic  accuracy  assessment  and  potentially  necessary  system  design  changes.  The  goal 
of  the  analysis  should  be  toward  designing  a  system  that  reflects  the  ITAS  tactical 
system’s  accuracy  as  closely  as  possible. 

The  initial  step  in  assessing  the  probability  of  a  scoring  event  centers  around 
defining  the  parameters  associated  with  miss  distance  from  the  TES  aimpoint.  Each  of 
the  major  biases  described  above  will  contribute  to  the  cumulative  mean  and  standard 
deviation  of  miss  distance  in  the  normal  training  engagement.  It  is  generally  accepted 
that  these  biases  follow  a  bivariate  normal  distribution  in  direct  fire  engagements.  [Ref 
11]  It  is  also  pmdent  to  assume  that  these  normally  distributed  biases  are  independent. 
Hence,  in  the  worst  case  scenario,  biases  would  be  additive  in  their  effects  upon  miss 
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distance  from  an  aimpoint.  Information  regarding  applicable  biases  from  previous  testing 
efforts  is  available  from  the  U.  S.  Army  Materiel  Systems  Analysis  Agency  (AMSAA) 
for  a  variety  of  systems.  [  Ref.  1 1]  Such  sources  of  information  can  be  useful  in  making 
assumptions  regarding  estimators  of  parameters  such  as  boresight  retention  variance. 
Once  the  variance  values  of  “interesting  biases”  are  determined,  either  through 
experimentation  or  assumption  based  upon  similar  system  performance,  a  variety  of 
simple  probability-of-hit  equations  can  be  employed.  One  alternative  model  follows. 

1.  Probability-Of-HU  For  Circular  Target  [Ref.  11] 

The  model  in  Equation  1  assumes  that  the  target  vehicle  or  impact  area  can 
be  modeled  as  a  circular  target.  In  this  scenario,  the  circular  target  assumption  presents 
no  particular  concern  as  we  are  viewing  the  target  as  a  thermal  image  centered  aroimd  a 
“hot-spot”  chosen  by  the  IT  AS.  The  formula  used  in  this  calculation  is  as  follows: 

(Equation  1)  P(hU)  =  1  - 

where  R  =  target  radius  and  a  represents  the  standard  deviation  or  bias  associated  with 
the  bivariate  normal  distribution.  Using  this  equation  requires  that  we  possess  the 
cumulative  aimpoint  error  estimates  for  our  system.  This  will  occur  only  if  miss  distance 
data  can  be  gathered  from  the  system.  An  example  of  a  simulated  probability-of-hit  at  a 
given  range  using  assumed  bias  values  follows,  where  R  =  1  meter  and  a  =  .5  meter 
which  is  equivalent  to  a  .167  mil  error  in  aimpoint  at  3000  meters.  The  calculation 
results  in  a  P(hit)  =  .865. 
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In  this  simple  approach  to  establishing  probabilities  of  hit,  we  use  the  “mil 
relationship”  to  establish  the  maximum  aimpoint  error  allowed  at  a  given  range. 
Aimpoint  errors  are  normally  measured  in  milliradians,  hence  the  relationship  that  1 
milliradian  =  1  meter  in  missed  distance  at  1000  meters  becomes  the  basis  for 
establishing  maximum  aimpoint  error.  In  the  example  above,  the  requirement  to  impact  a 
target  1  meter  in  radius  at  a  distance  of  3000  meters  equates  to  .333  milliradians 
maximum  aimpoint  error  at  the  time  of  firing.  Hence,  a  modest  cumulative  bias  figure  of 
.05  milliradians  in  aimpoint  error  translates  to  .15  meters  in  miss  distance  potential  at 
3000  meters.  The  example  calculation  above  provides  a  more  realistic  scenario,  yet  still 
points  out  that  a  .95  probability  of  scoring  event  is  unlikely. 

The  current  inability  to  define  the  aimpoint  of  the  BLT  in  relation  to  the 
ITAS  tactical  system  aimpoint  from  the  diagnostic  terminal  prevents  the  collection  of 
meaningful  data  regarding  the  bias  values  required  for  the  probability  of  hit  calculations 
above.  Contractor  efforts  to  enhance  engagement  results  were  primarily  restricted  to 
archaic  means  such  as  repeated  ITAS  boresighting,  gunner  aim-off  and  repeated  BLT 
system  boresighting.  Following  this  practice  establishes  a  new  accuracy  baseline  with 
each  system  adjustment  thus  sacrificing  repeatability  of  testing  or  any  hope  of  identifying 
dominant  system  biases. 

2.  Multiple  Regression  / ANOVA  Approach:  [Ref.  12] 

This  approach  relies  on  the  capability  to  measure  the  impact  of  the  MILES 
laser  on  a  target.  It  also  requires  the  ability  to  measure  the  errors  associated  with  the 
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major  bias  categories  selected  for  analysis.  The  ability  to  measure  boresight  retention  and 
gunner  aim  error  (independent  biases)  could  be  regressed  against  the  miss  distance 
measurement  at  the  target  (dependent  outcome).  Through  this  AA^alysis  Of  FAriance 
approach,  correlation  among  the  independent  biases  and  the  dependent  outcome  could  be 
assessed  and  lead  to  allocation  of  effort  in  troubleshooting  the  system. 


3.  Hypothesis  Testing:  [Ref.  12] 

Using  sufficient  data  collected  from  engagements  where  the  test  conditions 
have  remained  unchanged,  hypothesis  testing  could  be  conducted  using  the  test  statistic 
found  in  Equation  2. 


(Equation  2) 


where  x  is  derived  through  experimentation  and  the  remaining  parameters  relating  to 
population  means  po,  population  variance,  a,  and  sample  size  n  are  known  through  the  use 
of  historical  data  such  as  that  available  through  AMSAA.  Having  arrived  at  this  stage  of 
data  analysis,  one-sided  hypothesis  testing  could  be  accomplished  based  upon  a  null 
hypothesis  Ho  that  the  mean  aimpoint  error  is  less  than  .333  milliradians  in  the  case  of  a 
3000  meter  range  to  target.  An  expanded  example  of  hypothesis  testing,  using  .33  mils  as 
the  assumed  accuracy  requirement,  is  provided  in  the  next  section. 

Similar  calculations  using  the  test  statistic  above  in  conjunction  with  an 
established  a,  or  acceptable  Type  I  error  level,  can  be  used  to  provide  the  appropriate 
sample  size  for  experimentation  by  solving  for  n.  This  a  error  level  is  the  degree  to 
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which  the  government  is  willing  to  accept  a  system’s  mean  accuracy  as  valid,  when  in 
fact  the  system’s  true  accuracy  is  something  less.  In  this  thesis,  all  a  levels  are  at  the  .05 
level  and  all  confidence  intervals  are  at  the  95th  percentile.  Translated,  this  means  the 
government  will  accept  a  system  with  accuracy  less  than  it  wants  five  times  out  of  100. 
Or,  in  confidence  interval  terms,  95  times  out  of  100  the  tme  accuracy  of  the  system  will 
be  within  the  interval  established  through  testing.  The  difficulty  in  executing  hypothesis 
testing  lies  in  the  fact  that  current  test  design  does  not  permit  data  on  aimpoint  error  or 
miss  distance  to  be  gathered.  Without  redesigning  the  system  so  that  this  data  can  be 
obtained,  unfounded  assumptions  regarding  the  values  of  these  parameters  must  be  made. 

4.  Binomial  Distribution  Assessment:  [Ref.  13] 

Up  to  this  point,  alternatives  to  assessing  accuracy  have  focused  upon  the 
ability  to  make  assumptions  or  gather  data  pertaining  to  the  major  biases  affecting  system 
accuracy.  Yet  another  approach  to  accuracy  assessment  would  be  to  view  the  probability 
of  a  scoring  event  as  a  binomial  event.  Using  data  similar  to  that  found  in  Table  1,  a 

A 

value  for  the  parameter  p  could  be  derived  as  an  estimate  for  the  population  parameter  p 
(probability  of  successful  scoring  event).  Once  in  possession  of  this  estimator  ofp,  tables 
such  as  those  found  below  could  be  used  to  derive  a  confidence  interval  around  the 
desired  value  of  p.  For  example,  if  the  observed  engagement  success  rate  after  firing  20 
similar  engagements  is  .85,  using  Table  2,  we  derive  a  95%  confidence  interval  that  the 
trae  value  of  p  lies  between  .60  and  .96.  Under  this  scenario,  the  developer  would  not  be 
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pleased  with  the  results  nor  confident  that  the  tested  system  possesses  a  true  probability 
of  hit  near  .95. 

5.  Nonparametric  Analysis: 

Yet  another  option  in  assessing  the  quantitative  accuracy  of  the  PGTS  TES 
system  would  be  through  a  non-parametric  approach.  The  methods  applied  through  non¬ 
parametric  statistics  provide  a  simpler  means  of  hypothesis  testing  through  their  disregard 
of  population  parameters  or  their  underlying  distributions.  Since  we  believe  that  most  of 
the  biases  are  distributed  normally,  the  use  of  non-parametrics  in  this  instance  is  not 
necessary.  The  use  of  this  approach  indicates  lack  of  knowledge  regarding  the 
distribution  of  the  bieises  impacting  upon  system  accuracy.  The  nonparametric  approach 
is  not  a  contributor  to  confident  decision  making  when  other,  parametric,  methods  are 
applicable. 
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Table  2:  A  table  that  depicts  confidence  intervals  for  probability-of-hit, 
p,  in  binomial  sampling.  Experimental  data  is  used  to  develop  the  estimator 
forp  known  as  p  (derived  from  the  sample  fraction  of  #  hits  /  #  shots).  Based 
upon  the  sample  size  n,  a  confidence  interval  is  read  from  the  appropriate 
curves  in  the  table.  The  source  of  this  table  is  Reference  13. 


The  steps  involved  in  taking  a  statistical  based  approach  to  testing  are  not  as 
complex  as  they  might  appear.  In  the  case  of  the  TES  system,  execution  of  the  following 
steps  will  lead  to  a  more  quantitative  result  in  the  final  test  report.  A  recommended 
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detailed  testing  approach  is  provided  here  with  a  scenario  based  upon  this  approach  in  the 
following  section. 

The  first  step  in  analyzing  system  accuracy  is  to  choose  the  biases  impacting 
system  accuracy.  Not  all  of  these  biases  can  be  measured,  therefore  selection  of  those 
biases  largest  in  magnitude  is  preferred.  In  the  case  of  the  TES  system,  let  us  assume  that 
BLT  boresight  retention,  cumulative  HAS  bias  that  is  passed  from  that  system  and  cannot 
be  resolved  by  the  TES  contractor,  and  gunner  aim  error  are  the  potentially  greatest 
contributors  to  system  inaccuracies.  The  analysis  can  now  focus  upon  these  variables  as 
the  ones  causing  the  greatest  system  inaccuracies.  Table  3  provides  an  example  of  the 
data  values  that  might  be  generated  through  this  approach.  The  three  sets  of  bias  values 
in  Table  3  were  randomly  generated  via  a  normal  distribution  with  mean  equal  to  zero 
and  standard  deviation  equal  to  .20.  These  parameter  values  were  chosen  through 
experimentation  so  that  realistic  bias  values,  measured  in  mils,  could  be  used  in  Section  E 
of  this  chapter. 
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Cum  iTAS  Bias 


BLT  boresight 
retention 


Gunner  Aim/ 
Track  Error 


Abs  vaiue 
Cum  error 


Miss  Distance  & 
3000  meters^ 
3*CUM  ERROR 


MEAN 

0.260 

VAR 

0.026 

0.038 

0.038 

0.047 

STDEV 

0.160 

0.194 

0.194 

0.217 

Hypothesis  Test  Calculations 

mean-u 

-0.073 

Ho:  true  mean  <  .33  mils 

stdev/sqrt(n) 

0.040 

Ha:  true  mean  >=  .33  mils 

test  statistic 

-1.837 

95%  Confidence  Interval  Calculations 

Confidence  Interval  becomes: 

signifigance  value 

.260+/-1 .96*(sigma/sqrt(n)) 

.22<true  mean<.34 

-.034 

AAuic  jLfiapiajd  me  mtui  lu  auppuri  uic  it»ung  scenano.  Au  oias  values  were  ranaomiy 

generated.  The  results  indicate  a  system  of  acceptable  accuracy  through  hypothesis  testing  and 
development  of  a  confidence  interval. 

Next,  an  effort  should  be  made  to  conduct  engagements  that  strive  to  hold  one  of 
the  biases  constant.  This  step  will  again  require  instrumentation  that  enables 
measurements  to  be  taken  of  the  biases  described  in  Section  A  of  this  chapter  and  the 
miss  distance  at  the  target.  The  diagnostic  terminal  cited  in  the  Appendix,  with 
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cooperation  from  the  ITAS  contractor,  must  be  modified  to  determine  the  aimpoint 
variations  and  the  boresight  retention  rates.  The  ability  to  measure  system  aimpoint 
errors,  gunner  errors  and  miss  distance  will  prove  invaluable  as  shown  in  later  examples. 
It  may  be  necessary  to  make  assumptions  regarding  cumulative  errors  being  received  by 
the  ITAS  if  the  UAS  error  budget  is  not  available.  Having  established  acceptable 
methods  for  collecting  pertinent  data,  experimentation  is  conducted  at  various  ranges. 
Repeated  iterations  of  this  process,  each  time  varying  only  one  bias,  results  in  a  factorial 
design  that  leads  to  assessing  which  bias  impacts  accuracy. 

The  modified  instrumentation  used  to  collect  data  on  each  of  the  selected  biases 
during  this  process  will  provide  estimators  of  variation  and  mean  for  each  of  the  biases 
not  held  constant.  The  resulting  variability  information  can  lead  to  potential  system 
design  change  that  may  tighten  the  variation  leading  to  inaccuracy.  As  witnessed  at 
Redstone  Arsenal,  the  ability  to  conduct  numerous  engagements  is  not  a  problem, 
therefore  sample  size  is  a  minor  issue. 

The  final  step  is  to  evaluate  the  gathered  data,  such  as  that  in  Table  2.  Hypothesis 
testing  is  but  one  approach  that  can  be  used  to  determine  whether  the  system  performs  as 
well  as  required.  Additional  approaches  outlined  in  this  chapter  could  also  be 
implemented.  If  the  means  to  measure  miss  distance  at  the  target  is  available,  a 
regression  of  bias  values  against  miss  distance  could  be  conducted.  This  could  provide 
insight  into  variables  most  impacting  accuracy.  The  regression  approach  is  not  shown  in 
the  following  example  due  to  the  use  of  the  random  data  in  Table  2  from  known 
distributional  parameters.  The  results  would  not  be  interesting. 
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C.  A  DATA-BASED  SCENAMO 


To  reiterate,  the  values  introduced  in  Table  3  are  the  result  of  a  random  number 
generation  of  thirty  bias  values  for  the  three  largest  contributors  of  system  inaccuracy. 
The  size  of  the  sample  is  arbitrary  in  this  case.  One  method  to  analytically  choose  this 
sample  size  would  be  to  observe  that  in  Table  1  we  achieved  overall  P(hit)  of 
approximately  .72.  A  standard  rule  of  thumb  is  to  ensure  that  this  estimator  of  mean 

A 

accuracy  (p),  times  the  sample  size  (n),  exceeds  five  (i.e.  np  >=  5).  In  this  scenario  we 
have  30  times  .72,  or  approximately  21.  This  provides  confidence  that  the  sample  size  of 
30  used  in  this  scenario  is  sufficient.  The  data  used  in  Table  3  is  only  intended  to  assist  in 
describing  the  analysis  involved  in  the  proposed  testing. 

The  null  hypothesis  that  the  true  mean  accuracy  of  the  TES  system  is  less  than  or 
equal  to  (<=)  .33  mils  begins  the  hypothesis  testing  upon  the  data.  The  test  statistic 
shown  in  Table  3  indicates  a  “z-value”  of  -1.837.  This  would  indicate,  through  simple 
use  of  normal  tables,  that  there  exists  a  significance  value  of  approximately  .034  in  this 
case.  This  significance  value  means  there  are  approximately  3.4  chances  in  1,000  that  the 
trae  mean  accuracy  of  the  TES  is  worse  than  .33  mils.  The  government  would  therefore 
accept  the  system  if  its  criterion  was  significant  at  .05  or  better.  Remember,  the  .33  mils 
used  here  is  only  a  suggested  criterion  for  accuracy  that  permits  a  target  one  meter  in 
radius  to  be  hit  at  3000  meters.  There  are  many  that  do  not  feel  hypothesis  testing  is 
appropriate  in  this  application.  One  academic  source  reports,  “Significance  testing  in 
general  has  been  a  greatly  overworked  procedure,  and  in  many  cases  where  significance 
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statements  have  been  made  it  would  have  been  better  to  provide  an  interval  within  which 
the  true  value  of  the  parameter  would  be  expected  to  lie.”  [Ref.  13]  Using  the  same  data 
in  Table  2,  an  alternative  confidence  interval  is  easily  developed  using  Equation  3. 


(Equations)  3c-1.96* 


<;c<3c+1.96* 


This  provides  a  confidence  interval  for  TES  mean  accuracy  (x)  of  between  22 
mils  and  .34  mils.  The  result  shows  that  the  required  TES  accuracy  of  .33  mils  at  1000 


meters  being  used  in  this  scenario  is  within  the  range  of  the  95  percent  confidence 
interval  calculated.  In  fact,  the  required  accuracy  is  at  the  high  end  of  the  confidence 
interval  indicating  that  approximately  95  times  out  of  100  we  can  expect  the  true  mean 
TES  accuracy  to  be  better  than  .33  mils  within  the  above  interval.  This  provides  the  same 
indication  that  the  system  accuracy  using  the  simulated  data  is  acceptable. 

The  overall  technique  described  here  focuses  upon  the  major  variables  affecting 
accuracy  of  the  TES  system.  If  miss  distance  at  the  target  can  be  obtained,  a  comparison 
of  bias  values  to  the  response  variable  of  miss  distance  could  be  conducted.  Again,  the 
data  used  was  randomly  generated  under  a  set  of  conservative  assumptions.  True  data 
will  have  to  be  scrutinized  to  a  far  greater  degree  to  address  such  areas  as  outliers,  and 
other  factors  that  dirty  every  data  set.  The  example  does  highlight  the  relative  simplicity 
of  applying  the  tools  of  analysis  presented  once  the  pool  of  potential  inhibitors  to 
accuracy  has  been  narrowed.  The  example  also  points  out  the  benefits  of  establishing 
accuracy  via  a  measure  such  as  milliradians  vice  simply  stating  that  a  95%  probability  of 
a  scoring  event  must  occur.  In  fact,  it  is  recommend  that  miss  distance,  vice  probability 
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of  a  scoring  event,  become  the  measure  through  which  specification  compliance  is 
measured.  This  approach  to  testing  that  establishes  a  mil  relationship  to  accuracy  would 
permit  the  majority  of  testing  to  take  place  at  the  maximum  required  range  of  3750 
meters.  Although  the  shorter  ranges  outlined  in  the  Appendix  may  require  limited 
verification  of  accuracy,  once  the  system  has  proven  that  it  can  achieve  miss  distance 
tolerances  at  long  range,  the  ability  to  achieve  similar  results  at  shorter  ranges  is  assured. 
The  major  effort  in  implementing  this  testing  will  be  in  the  instramentation  necessary  for 
data  collection.  The  testing  evolution  observed  at  Redstone  Arsenal  would  remain 
virtually  unchanged,  with  the  exception  of  increased  testing  involving  the  movement  of 
the  system  between  engagements  to  verify  boresight  retention.  To  review,  a  short 
summary  of  the  critical  steps  to  improving  the  testing  process  are  as  follow: 

♦  Choose  the  biases,  largest  in  magnitude  that  impact  system  accuracy.  Not  all 
of  these  biases  can  be  easily  measured,  therefore  selection  of  those  biases  largest  in 
magnitude  will  provide  the  best  return  per  instmmentation  dollar  spent. 

♦  Conduct  engagements  that  strive  to  hold  one  of  the  biases  constant.  This  step 
allows  biases  impacting  accuracy  to  be  assessed  individually.  The  ability  to  measure 
system  aimpoint  errors,  gunner  errors  and  miss  distance  will  prove  invaluable  in 
establishing  inter-relationships  among  the  detractors  from  system  accuracy. 

♦  Using  a  detailed  data  collection  plan,  conduct  testing  that  is  repeatable  and 
returns  a  sufficient  database  with  which  to  perform  accuracy. 

♦  Attempt  to  derive  measures  of  central  tendency  and  standard  deviation  for  each 
of  the  biases  of  interest. 
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♦  Conduct  analysis  upon  the  data  as  described  in  the  testing  scenario  in  Section 
E  of  this  chapter. 

♦  Finally,  be  prepared  to  explain  and  defend  the  results  of  the  analysis.  The 
quality  of  procurement  decision  will  ride  on  the  quality  of  the  analysis  conducted. 

D.  CONCLUSIONS  PERTAINING  TO  TES  ACCURACY 

At  present,  accuracy  of  the  TES  system  cannot  be  determined.  The  approach  to 
testing  currently  being  employed  in  PGTS  development  falls  short  of  providing 
meaningful  quantitative  results.  The  lack  of  attention  to  accuracy  in  employing  the 
MILES  system  explains  an  underlying  cause  of  a  historical  lack  of  confidence  by  the  user 
in  the  MILES  system.  In  order  to  achieve  realistic  results  in  the  training  environment,  the 
development  of  the  training  system  must  mirror  the  development  of  the  tactical  system. 
Since  the  MILES  burst  laser  transmitter  is  wide-beamed  by  nature,  the  accuracy  of  the 
aimpoint  of  that  laser  is  often  discounted.  This  results  in  poor  “kill  rates”  as  the 
periphery  of  the  beam  is  often  that  which  is  engaging  the  target.  This  advance  in 
technology  must  address  these  shortcomings  in  order  to  be  accepted  as  a  viable  training 
device  in  the  fleet.  The  method  of  engaging  a  High  Mobility  Multi-Purpose  Wheeled 
Vehicle  (HMMWV)  at  various  ranges,  a  few  times,  under  a  variety  of  conditions  does  not 
establish  the  PGTS  as  a  realistic,  accurate  training  device  suitable  of  emulating  the  TOW 
ITAS  system. 

It  is  the  author’s  belief  that  the  developer  of  the  PGTS  is  approaching  the  system 
accuracy  question  from  a  disadvantageous  position.  Without  possessing  the  boresight 
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information  from  the  HAS,  or  any  capability  of  influencing  the  dispersion  of  the  MILES 
laser,  the  contractor  should  concern  himself  with  achieving  the  system  aimpoint 
established  by  the  IT  AS.  Once  this  effort  is  achieved,  system  accuracy  is  out  of  the  hands 
of  the  IT  AS  developer.  Due  to  the  fact  that  the  PGTS  is  a  subsystem  of  the  HAS  tactical 
system,  the  developer  should  concentrate  his  efforts  upon  proving  that  his  BLT  aimpoint 
is  the  same  as  the  HAS  system  aimpoint  and  that  retention  of  this  concurrent  aimpoint  is 
maintained.  The  majority  of  this  effort  could  be  conducted  in  the  laboratory  environment 
and  would  require  modifications  to  the  diagnostic  system  that  extracts  information  from 
the  HAS  system.  Once  the  developer  has  established  that  his  BLT  system  consistently 
points  to  the  same  aimpoint  as  the  HAS  system,  little  else  can  be  accomplished  from  an 
accuracy  standpoint  other  than  designing  the  BLT  interface  with  the  HAS  so  that 
boresight  loss  is  minimized. 

Finally,  if  realism  is  to  be  preserved,  testing  must  be  expanded  to  include  analysis 
of  boresight  retention  while  moving  from  one  firing  position  to  a  subsequent  firing 
position.  Based  upon  observations  of  testing  at  Redstone  Arsenal,  this  is  an  area  of 
potential  accuracy  degradation  that  warrants  additional  research. 
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VI.  ENHANCED  INSTRUCTOR  STATION  (REHOST) 


A.  BACKGROUND 

The  Enhanced  Instructor  Station  (EIS),  commonly  referred  to  as  the  REHOST 
program,  is  designed  to  replace  the  currently  fielded  TOW  Gunnery  Trainer  (GT).  The 
REHOST  trainer  is  an  indoor  system  that  provides  training  to  TOW  gunners  through  the 
presentation  of  video  based  scenarios.  The  system  used  by  the  gunner  is  intended  to  be 
the  functional  equivalent  of  the  tactical  system.  During  the  course  of  gunner 
engagements,  the  instructor  has  the  capability  to  assess  gunner  performance  via  a  video 
display  that  is  identical  to  that  which  the  gunner  is  observing.  The  instructor  controls 
which  of  the  90  programmable  scenarios  that  the  gunner  will  be  facing.  Following  the 
engagement,  the  gunner  can  be  critiqued  on  a  variety  of  gunner  skills.  Student 
performance  appraisals  include  the  following  information: 

♦  Hit  or  Miss 

♦  Range  at  which  missile  was  lost 

♦  Ground  impact  location 

♦  Miss  distance  in  plane  of  target  (High,  Low,  Left ,  Right) 

♦  Gunner  aiming  error 

♦  Gunner  tracking  score  based  upon  sampling  of  aim  error  during  the  duration  of 
the  missile  flight. 

The  primary  puipose  of  the  REHOST  program  is  to  provide  improvements  to  the 
above  functional  aspects  of  the  system.  There  will  be  no  new  capability  until  a  future 
upgrade  is  provided  that  permits  instructor  programmable  scenarios. 
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Major  enhancements,  to  the  currently  fielded  version  of  the  REHOST  trainer,  provided 
under  this  contract  include: 

♦  A  Pentium®-based  processor 

♦  Increased  memory  capacity 

♦  A  future  capability  to  program  threat  scenarios  via  CD  ROM 

♦  User— friendly  Windows®— based  menus. 

The  effort  involved  in  executing  this  portion  of  the  PGTS  program  is  based  upon 
a  4  million  dollar  effort  for  five  prototypes  that  were  nearing  completion  at  the  time  of  the 
author’s  internship  with  the  prime  contractor.  Similar  to  all  efforts  involved  in  the  PGTS 
family  of  training  devices,  the  design  specification  for  the  REHOST  program  is  a  1994 
update  based  upon  the  original  1983  contract. 

B.  OBJECTIVE 

This  portion  of  the  thesis  is  devoted  to  an  analysis  of  the  REHOST  testing  plan. 
Similar  to  the  previously  discussed  TES  development  effort,  the  methodology  involved  in 
executing  the  testing  of  the  REHOST  program  is  presented  and  critiqued.  Additionally, 
an  analysis  of  the  improvements  inherent  in  the  REHOST  system  is  conducted  in  an 
effort  to  assess  the  magnitude  of  improvement  over  the  currently  fielded  system. 
Secondary  thesis  objectives  focus  upon  the  compatibility  of  this  training  device  with  the 
newly  developed  ITAS  system  and  the  training  value  of  the  REHOST  training  concept  in 
general. 
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C.  TESTING  THE  REHOST 


The  testing  effort  applied  to  the  REHOST  program  was  far  more  limited  than  the 
effort  applied  to  the  TES  program  discussed  earlier  in  this  thesis.  It  appeared  that  the 
view  of  the  designers  was  that  the  REHOST  was  only  an  upgrade  of  hardware  and 
software,  and  would  not  impact  the  existing  performance  of  the  trainer.  Due  to  this 
assumption,  a  Functional  Acceptance  Test  Plan  (FATP)  was  developed  directly  from  the 
1986  version  used  to  field  the  current  system. 

Since  the  REHOST  is  an  indoor  trainer,  test  design  requirements  are  significantly 
different  than  those  discussed  in  reference  to  the  TES  effort.  In  fact,  the  testing 
requirements  of  indoor  systems  are  normally  less  complex  than  those  of  outdoor  systems 
simply  because  many  of  the  items  referred  to  in  Chapter  IV  are  not  applicable.  This  fact 
does  not  relieve  the  parties  involved,  both  government  and  contractor,  of  ensuring  that 
complete  specification  compliance  is  achieved. 

The  approach  undertaken  to  assess  the  testing  of  the  REHOST  program  was 
similar  to  the  approach  taken  with  the  TES  effort.  First,  a  study  of  the  contract  under 
which  the  contractor  was  performing  was  conducted.  Then  the  test  procedures  that  would 
be  used  in  government  acceptance  testing  of  the  five  prototypes  were  reviewed.  The 
verification  of  the  FATP  was  conducted  by  the  author  as  an  assigned  task  through  the 
internship.  This  task  pennitted  an  in-depth  analysis  of  the  methodology  that  would  be 
applied  to  the  government  acceptance  testing  effort.  This  document  review  resulted  in 
181  discrepancies  ranging  from  differences  in  REHOST  functional  operation  to 
deficiencies  in  establishing  contract  compliance. 


45 


Unlike  the  TES  testing  effort,  no  FATP  effort  was  presented  indicating  that  the 
REHOST  system  would  be  tested  for  compliance  with  the  1994  REHOST  design 
specification.  [Ref.  7]  An  example  of  this  oversight  is  seen  in  the  lack  of  verification  of 
required  fields  of  view,  required  audio  levels,  and  required  gunner  tracking  sensitivity.  In 
its  current  form,  the  FATP  developed  for  the  REHOST  system  provides  an  operator’s 
manual  for  the  proper  functioning  of  the  system  vice  a  test  plan  that  adequately  verifies 
specification  compliance. 

Upon  completion  of  the  author’s  review  and  verification  of  the  FATP,  a  complete 
list  of  discrepancies  relating  to  the  system  and  to  test  methodology  were  submitted  to  the 
REHOST  systems  engineer  for  incorporation  into  the  draft  that  would  be  submitted  for 
government  approval.  Other  than  mmor  editorial  changes  and  limited  references  to 
functional  discrepancies,  the  FATP  was  subsequently  submitted  “as-is,”  for  government 
review.  The  results  of  the  government  review  were  not  available  at  the  time  of  release  of 
this  thesis.  However,  it  is  anticipated  that  a  thorough  government  review  of  this 
document  will  result  in  greater  emphasis  being  placed  upon  ensuring  REHOST 
specification  compliance.  It  also  must  be  noted  that  a  thorough  review  of  the  REHOST 
FATP  can  only  be  accomplished  with  access  to  the  system.  Any  government  review  of 
the  REHOST  test  plan,  without  access  to  the  system,  would  not  be  a  productive  effort. 

Through  the  process  of  reviewing  both  the  TES  and  the  REHOST  test  procedures, 
it  was  evident  that  there  existed  distinct  differences  in  the  approach  to  testing  between  the 
two  programs.  The  testing  documents  submitted  for  government  approval  were 
significantly  different  in  their  content,  detail,  and  their  approach  toward  achieving 
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specification  compliance.  It  was  anticipated  that  there  would  exist  some  degree  of 
commonality  among  program  documentation  designed  to  achieve  the  same  goal  of 
prototype  acceptance.  It  would  also  be  a  prudent  assumption  that  the  government 
program  oflQce  tasked  with  development  of  these  two  PGTS  programs  would  anticipate 
similar  program  testing  documentation.  As  an  outsider  working  with  these  programs,  it 
would  be  easy  to  come  to  the  conclusion  that  two  different  contractors  and  two  different 
government  agencies  were  involved  in  a  program  that  possesses  commonality  in 
personnel  from  the  program  manager  level  and  higher. 

D.  REHOST  REALISM 

The  development  of  the  REHOST  system,  as  with  all  of  the  PGTS  programs,  is 
designed  to  provide  realistic  TOW  gunnery  training  and  instruction.  While  the  TES 
effort  and  the  upcoming  PG  effort  meet  the  litmus  test  of  realism,  the  REHOST  program 
falls  drastically  short. 

The  student  station,  which  will  not  be  modified  under  this  effort,  does  not  provide 
functional  realism  with  the  currently  fielded  TOW  tactical  system.  Knobs  and  switches 
used  to  perform  training  are  different  in  their  location  and  function  from  the  current 
tactical  system  and  drastically  different  from  the  IT  AS  system  under  development.  This 
discrepancy  is  primarily  due  to  the  fact  that  the  student  station  does  not  utilize  an  actual 
TOW  tactical  system.  It  uses  a  system  that  is  essentially  a  box  housing  the  electronics 
necessary  for  a  gunner  to  view  the  preprogrammed  scenarios  chosen  by  the  instructor. 
This  is  an  understandable  limitation  given  the  technology  of  the  early  eighties.  It  is  not 
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understandable  in  light  of  the  existing  video  technologies  being  utilized  to  monitor  the 
TES  program. 

As  seen  with  the  ET,  TES  and  the  future  PG  family  of  PGTS  systems,  training  can 
be  incorporated  into  the  tactical  system.  All  of  the  training  conducted  with  these  three 
systems  will  utilize  the  same  equipment  being  used  by  TOW  gunners  imder  combat 
conditions.  Any  upgrade  to  the  training  system  being  developed  under  the  REHOST 
program  should  attempt  to  accomplish  a  similar  degree  of  realism. 

An  alternative  approach  would  be  to  modify  the  REHOST  program  so  that  it  is 
fielded  concurrently  with  the  ITAS  tactical  system  being  developed.  The  remainder  of 
the  PGTS  programs  are  following  this  path.  The  REHOST  program  modifications  could 
then  incorporate  the  same  philosophy  as  its  sister  programs.  That  philosophy  is  the 
utilization  of  the  same  hardware  being  used  in  the  field  as  the  basis  of  our  training 
devices.  Modifications  to  this  program  would  center  around  the  development  of  a  new 
instructor  station.  The  instructor  station  would  only  interface  with  the  tactical  system  in  a 
garrison  environment  or  in  the  case  of  trainees,  in  the  school  environment.  This  approach 
would  maintain  the  philosophy  that  the  best  training  value  is  gained  when  a  student  or  a 
user  operates  in  the  field  with  his  actual  combat  equipment  instead  of  a  training 
substitute. 

E.  FUTURE  RESEARCH  POTENTIAL 

There  is  question  that  the  REHOST  development  effort  serves  as  a  viable  training 
device  to  realistically  train  TOW  gunners.  Future  research  that  could  be  conducted  to 
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prove  or  disprove  this  statement  could  consist  of  an  effort  to  establish  a  relationship 
between  REHOST  type  training  engagement  scores  and  tactical  system  training  results. 


A  system  of  developing  validity  coefficients  that  show  a  correlation  between 
REHOST  training  and  tactical  training  would  be  conducted  to  ensure  that  REHOST 
scores  have  a  predictive  capability  of  field  performance.  A  regression  analysis,  similar  to 
that  proposed  in  Chapter  V  would  suffice.  If  REHOST  is  then  proven  to  provide 
predictive  potential  in  assessing  TOW  gunner  performance,  then  there  exists  potential  to 
select  the  most  qualified  the  trigger  men  for  the  TOW  missile  system.  If  REHOST  results 
cannot  be  correlated  to  field  performance,  one  must  assess  the  overall  viability  of  this 
training  device.  This  area  of  research  could  be  the  subject  of  a  follow-on  thesis.  Since 
TOW  missiles  currently  cost  approximately  $20,000  each,  substantial  savings  and 
potentially  increased  combat  effectiveness  could  result  from  further  research  in  this  area. 
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Vn.  PROGRAM  OBSERVATIONS 


^‘Providing  quality  products  needed  by  the  United  States  Armed 
Forces  requires  highly  disciplined,  yet  flexible  management  framework  that 
effectively  translates  operational  needs  into  stable,  affordable  acquisition 
programs.*^ [Ref.  8] 

Thus  far,  this  thesis  has  focused  upon  the  technical  aspects  of  the  PGTS  program. 
In  particular,  emphasis  has  been  placed  upon  the  test  methodology  adopted  by  the  TES 
and  REHOST  programs  being  prepared  for  final  government  acceptance  of  the  prototype 
systems. 

This  thesis  now  addresses  programmatic  issues  involving  both  programs  that^  if 
improved,  will  enhance  the  quality  of  these  two  systems.  Unlike  the  technical  aspects  of 
testing  design  specifications,  it  is  more  difficult  to  apply  operations  research  skills 
directly  to  those  areas  of  program  management  that  are  less  technical.  However,  areas 
that  can  be  addressed  by  operations  research  skills  will  serve  as  quantitative  input  to  the 
program  that  will  contribute  to  any  effectiveness  analysis  of,  in  this  case,  the  PGTS 
systems. 

One  of  the  major  areas  of  emphasis  in  the  current  5000  series  DoD  acquisition 
document  points  out  the  importance  of  Integrated  Product  Teams  (PTs)  at  various  levels. 
[Ref.  1]  The  concept  of  integrated  management  teams  is  not  a  cliche  intended  to  bring 
current  programs  into  21st  century  management  style.  It  is  a  concept  that  promotes 
sharing  of  ideas  and  challenges  among  program  participants.  The  output  from  integrated 
product  teams  will  be  measures  of  program  effectiveness,  or  metrics,  that  indicate 
program  efficiency  [Ref  8] 
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During  seven  weeks  of  internship  at  the  prime  contractor’s  facility,  concerns 
surrounding  the  focus  of  the  REHOST  program  developed.  A  common  focus  relating  to 
achieving  contract  success  was  no  longer  apparent  as  program  schedule  concerns 
dominated  a  majority  of  the  effort.  The  utilization  of  an  integrated  product  team  that 
brought  the  government  developer  and  the  contractor  together  may  have  focused  efforts 
in  a  more  appropriate  direction  by  providing  immediate  schedule  relief.  Additional 
benefits  may  have  been  realized  in  reducing  risk  by  eliminating  the  development  of 
overly  ambitious  revised  program  schedules. 

As  a  result,  recommendations  centered  around  potential  enhancements  in  program 
execution  were  made  with  the  intent  of  providing  an  independent  assessment  from 
someone  not  involved  in  the  daily  details  of  the  program.  Some  of  the  major  areas  cited 
for  potential  improvement  are  as  follow: 

Training:  Advantage  must  be  taken  of  the  myriad  of  government  procurement 
training  opportunities.  These  range  in  complexity  from  the  DSMC  program  manager’s 
course  to  short  courses.  Normally,  contractor  participation  is  welcome  as  it  tends  to 
foster  the  govemment/industry  team  concept.  At  any  rate,  participation  will  reduce  the 
unknowns  of  government  expectations. 

Historical  Reference  to  Other  Programs:  The  development  of  contract  documents 
and  the  execution  of  program  functions  such  as  developmental  testing  are  much  easier 
when  a  search  of  the  lessons-leamed  in  other,  similar,  programs  is  conducted.  This 
concept  extends  to  discussions  among  project  managers  in  trading  ideas  across  current 
programs.  This  free-flowing  information  exchange  has  significant  implications  in  the 
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Quality  Assurance  arena.  It  was  anticipated  that,  through  the  experience  of  participating 
in  the  testing  at  Redstone  Arsenal,  contractor  Quality  Assurance  representatives  were 
charged  with  providing  an  independent  assessment  of  program  activities.  In  order  for  this 
function  to  be  carried  out  most  effectively,  training  of  personnel  in  the  skills  of 
quantitative  analysis  seems  prudent.  If  trained  in  operations  research  techniques,  it  would 
appear  that  the  Quality  Assurance  organization  in  any  program  could  serve  as  the  honest 
broker  of  applying  those  techniques  to  program  development. 

Defining  the  Critical  Path:  Establishing  the  critical  path  of  program  events, 
whether  manufacturing  related  or  testing  related,  will  assist  in  meeting  schedule  deadlines 
and  reducing  program  risk.  Current  REHOST  scheduling  documents  do  not  highlight 
critical  path  program  events  that  should  be  weighted  more  heavily  in  the  event  that  those 
items  impact  other  program  events.  An  example  is  software  loading  into  the  upgraded 
processor  that  must  be  accomplished  prior  to  commencing  contractor  validation  of  test 
procedures.  Definition  of  the  critical  path  provides  the  primary  tool  for  prioritization  of 
program  resource  allocation. 

Risk  Assessment:  Risk  must  be  prioritized,  and  based  upon  this  prioritization, 
appropriate  program  resources  applied.  In  the  circumstances  surrounding  the  REHOST 
program,  the  risk  associated  with  continuing  program  execution  versus  temporarily 
halting  the  allocation  of  resources  that  may  be  applied  to  other  programs  is  worthy  of 
research.  Risks  associated  with  continuing  a  program  with  significant  schedule  and 
design  challenges  will  likely  lead  to  procurement  of  a  system  of  lesser  quality  than 
originally  intended.  Until  the  true  training  value  of  the  REHOST  program  or  the  accuracy 
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of  the  TES  system  is  established,  both  of  these  programs  are  in  a  high  risk  state. 

Clearly  Defined  Data  Requirements:  At  some  point  a  program  must  address  the 
minimum  data  requirements  to  validate  system  performance.  The  method  of  testing 
based  solely  upon  hit  or  miss  results  is  too  limited  in  scope.  Enhancements  to  current 
data  collection  efforts  would  include  such  items  as  system  operating  hours,  laser  firings, 
range  conditions  and  the  data  items  previously  discussed  in  Chapters  IH  through  V. 
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Vm.  RECOMMENDATIONS 


At  present,  it  is  not  possible  to  determine  the  accuracy  of  the  TES  system  or  the 
training  value  of  the  REHOST  program  through  quantitative  analysis.  This  fact  should 
result  in  a  significant  amount  of  concern  by  the  program  manager  and  the  procurement 
decision  maker.  A  thorough  review  of  the  method  being  utilized  to  establish  accuracy  of 
the  TES  program  and  future  research  into  the  training  value  of  the  REHOST  system  will 
remedy  this  situation. 

The  current  specification  compliance  methodology  being  applied  to  the 
acceptance  of  both  REHOST  and  TES  systems  does  not  support  a  valid  verification  that 
these  systems  will  perform  in  the  manner  which  is  expected  under  current  contracts.  Any 
procurement  decision  made  based  upon  the  results  of  testing  to  date  would  have  to  infer 
that  the  TES  system  is  not  capable  of  achieving  a  .95  probability  of  achieving  a  scoring 
event  at  any  range.  The  more  thorough  statistical  approach  presented  in  Chapters  IH 
through  V  would  provide  quantitative  information  capable  of  establishing  current  system 
accuracy.  The  procedures  for  thorough  testing  are  in  place  in  the  TES  program,  but  a 
significant  enhancement  to  data  collection  must  take  place. 

The  REHOST  program  must  begin  with  abandoning  the  concept  that  the 
modifications  being  made  to  the  former  system  are  essentially  cosmetic  and  therefore  do 
not  require  thorough  validation  of  contract  specifications.  A  test  plan  that  thoroughly 
validates  all  contract  specifications  is  warranted  when  the  subject  system  is  being 
procured  at  the  prototype  cost  of  $800,000  per  copy.  It  is  also  reasonable  to  assume  that 
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the  contractor  views  these  modifications  as  substantial  due  to  the  fact  that  in  excess  of 
100  hours  of  contractor  effort  has  been  applied  to  the  primary  testing  document  for  the 
REHOST  system.  This  level  of  effort  indicates  a  desire  to  thoroughly  test  REHOST, 
however,  the  FATP  being  submitted  for  government  review  in  December,  1995,  did  not 
achieve  that  goal.  The  primary  question  is,  if  this  program  does  not  substantially  modify 
the  existing  gunnery  system,  then  does  it  provide  value  in  training  our  forces? 

The  training  value  resident  in  the  TES  system  is  undeniable.  It  permits  a  TOW 
gunner  to  conduct  training  in  the  field  at  any  time  when  in  the  presence  of  his  system. 
The  ability  to  conduct  this  type  of  training  in  a  flexible  fashion,  at  the  lowest  unit  levels, 
will  prove  invaluable.  Most  importantly,  it  conducts  the  training  with  the  very  same 
equipment  that  will  be  used  in  the  event  of  conflict. 

As  indicated  in  Chapter  IV,  the  validation  of  training  value  in  the  REHOST 
program  requires  further  research.  An  expanded  study  that  establishes  the  correlation 
between  REHOST  training  performance  and  TOW  gunnery  skills  in  the  field  should  be  a 
prerequisite  to  procurement. 

In  closing,  the  reader  is  reminded  that  this  thesis  is  not  intended  to  intimate  that  an 
acceptable  level  of  commitment  is  not  resident  in  the  POTS  program,  rather,  it  is  intended 
to  provide  an  operations  research  perspective  to  program  management  that  may  lead  to 
more  effective  analysis  of  program  development  through  the  ability  to  quantify,  where 
possible,  factors  impacting  system  performance.  The  contractor’s  commitment  to 
excellence  is  self-evident  through  his  enthusiastic  support  of  the  internship  that  served  as 
the  basis  for  this  work. 
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APPENDIX 


SELECTED  TES  TRAINER  TEST  PROCEDURES 
This  appendix  is  provided  so  that  the  reader  may  follow  in  detail,  the  testing 
procedures  implemented  during  TES  program  development  and  to  contrast  them  with  the 
test  methodology  presented  in  this  thesis.  It  is  also  provided  as  ready  reference  to  the 
issues  discussed  in  Chapters  III  through  V  of  the  thesis.  This  section  of  the  Test 
Procedures  provides  a  complete  chronology  of  the  events  that  each  of  the  prototype 
systems  would  be  subjected  to  at  Redstone  Arsenal  during  government  prototype 
acceptance  testing. 

Attention  is  drawn  to  the  limited  testing  procedures  relating  to  the  accuracy  of  the 
TES  system  and  the  lack  of  testing  procedures  verilying  boresight  retention  following 
movement  of  the  HMMWV  mounted  TES  system.  The  annotations  shown  in  the 
Appendix  represent  the  final  editing  effort  prior  to  government  acceptance  testing. 
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